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CHAPTER  I 


INTRODUCTION 


Preface 

The  DOD  acquires  major  weapon  systems  through  con¬ 
tractual  efforts  between  the  government  and  civilian  corpora¬ 
tions.  As  a  major  weapon  system  progresses  through  the 
acquisition  life  cycle  and  enters  the  production  and  deploy¬ 
ment  phase,  design  deficiencies  will  surface  which  require 
correction.  The  correction  of  these  deficiencies  is  gener¬ 
ally  accomplished  through  the  implementation  of  an  engineer¬ 
ing  change  proposal  (ECP)  which  has  been  approved  by  the 
configuration  control  board  of  the  responsible  system  program 
office.  The  implementation  of  an  ECP  requires  modification 
of  the  contract  because  the  change  contains  "terms  [which 
are]  different  from  those  which  were  agreed  upon  in  the  ori¬ 
ginal  contract  [4:87]."  There  are  two  contractual  tools 
available  to  modify  the  contract  under  the  authority  of  the 
change  clause;  these  are  the  unilateral  change  order  and  the 
bilateral  supplemental  agreement. 


Literature  Review 


Topic 

This  literature  review  addresses  factors  to  consider 
in  the  use  of  the  change  order  and  the  supplemental  agreement. 
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This  review  emphasizes  the  cost  and  schedule  characteristics 
of  each  method. 

Key  Terms 

1.  Advanced  Change  Study  Notice  (ACSN)  --  A  document 
(AFSC  Form  223)  which  precedes  a  preliminary  or 
formal  ECP  (or  contract  change  proposal)  and 
which  contains  information  establishing  the  need 
for  the  change.  It  is  used  to  allow  effective 
initial  evaluation  of  the  suggested  change 
[13:84]. 

2.  Ceiling  Price  --  This  is  the  dollar  amount  at 
which  the  government  obligation  ceases.  When 
contract  cost  reaches  this  point,  the  contrac¬ 
tor's  profit  is  zero  dollars.  Any  expenditure 
beyond  the  ceiling  price  must  be  absorbed  by  the 
contractor  from  sources  not  obligated  on  the 
instant  contract  under  consideration  [10:133]. 

The  term  ceiling  price  is  peculiar  to  fixed 
price  incentive  contracts  [9]. 

3.  Change  Order  (C/O)  —  A  written  order  signed  by 
the  contracting  officer,  directing  the  contrac¬ 
tor  to  make  changes  which  the  changes  clause  of 
the  contract  authorizes  the  contracting  officer 
to  order  without  the  consent  of  the  contractor 
(often  referred  to  as  "unpriced  actions")  [12:1: 
15]  . 

4.  Configuration  Control  Board  (CCB)  --  A  board 
composed  of  representatives  from  program/project 
functional  areas  such  as  engineerina,  configura¬ 
tion  management,  procurement,  production,  test 
and  logistics  support,  training  activities  and 
using/supporting  organizations.  This  board 
approves  or  disapproves  proposed  change  requests 
with  each  member  recording  his  organization's 
official  position  [on  the  ECP],  The  program/ 
project  manager  is  normally  the  board  chairman 
and  makes  the  final  decision  on  all  chancres 
[13:85]  . 

5.  Contract  Modification  --  Any  written  alteration 
in  the  speci f ication (s ) ,  delivery  point,  rate 
of  delivery,  contract  period,  price,  quantity, 
or  other  contract  provisions  of  an  existing  con¬ 
tract,  whether  accomplished  by  unilateral  action 
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in  accordance  with  a  contract  provision,  or  by 
mutual  action  of  the  parties  to  the  contract. 

It  includes  (i)  bilateral  actions  such  a.  sup¬ 
plemental  agreements,  and  (ii)  unilateral  actions 
such  as  change  orders,  orders  for  provisioned 
items,  administrative  changes,  notices  of  termi¬ 
nation,  and  notices  of  the  exercise  of  a  contract 
option  [12:1:15] . 

6.  Def initization  --  The  process  by  which  the  gov¬ 
ernment  and  the  contractor  agree  to  the  specific 
tasks  to  be  performed  under  the  scope  of  a  con¬ 
tract  modification  and  determine  the  negotiated 
price  associated  with  those  tasks  [9] . 

7.  Engineering  Change  Proposal  (ECP)  --  A  formal, 
priced  document  (DD  Forms  1692)  used  to  propose 
changes  to  the  contract  provisions  and  scope... 
especially  when  related  equipment,  (configura¬ 
tion  items) ,  interfaces  or  technical  manuals  are 
affected  or  retrofit  is  involved  [13:86]. 

8.  Not-to-exceed  (NTE)  price  —  A  not- to-exceed 
price  serves  the  same  purpose  for  change  orders 
that  a  ceiling  price  does  for  fixed  price  incen¬ 
tive  contracts.  Specifically,  the  NTE  price 
caps  the  government's  liability  if  the  contractor 
is  performing  an  unpriced  effort.  Any  expendi¬ 
ture  beyond  the  NTE  price  must  be  absorbed  by 
the  contractor  [9]. 

9.  Price  Adjustment  Factor  --  A  term  used  exclusive¬ 
ly  in  this  thesis  effort,  the  price  adjustment 
factor  is  the  difference  between  the  not-to- 
exceed  price  submitted  with  an  unpriced  proposal 
and  the  final  def ini ti zed  price  of  the  ECP 
expressed  as  a  percentage  of  the  original  NTE. 


10.  Production  Effectivity  —  The  sequential  number 
associated  with  the  first  system  to  receive  a 
modification  in  production. 

11.  Program  Manager  (PM)  --  The  single  Air  Force 
Manager  (system  program  director,  program  or 
project  manager,  or  system  or  item  manager) 
during  any  specific  phase  of  the  acquisition 
life  cycle  [14:96]. 

12.  Supplemental  Agreement  —  Any  contract  modifi¬ 
cation  which  is  accomplished  by  the  mutual 
action  of  the  parties  [12:1:18]. 

Discussion 


The  literature  reviewed  for  this  research  effort  in¬ 
cluded  many  periodicals,  government  publications ,  regulations, 
and  Department  of  Defense  sponsored  research  reports.  The 
information  reported  in  this  discussion  has  been  extracted 
from  Department  of  Defense  sponsored  research  projects  which 
address  contract  modifications. 

The  literature  reviewed  in  this  study  discussed  six 
factors  to  consider  when  incorporating  an  ECP  by  change  or¬ 
der.  The  first  factor,  reported  by  McMaster  in  A  Procurement 
Guide  for  Special  Projects  Officers  is  a  loss  of  cost  control 
when  a  modification  is  incorporated  by  change  order. 

McMaster  explains  this  characteristic  of  the  change  order  as 
follows : 


Urgent  changes  in  the  requirements  can  be  incor¬ 
porated  by  a  change  order  prior  to  price  negotia¬ 
tion  provided  a  ceiling  price  for  the  changes  has 
been  received  by  the  contracting  officer  from  the 
contractor.  This  ceiling  price  is  usually  pre¬ 
pared  in  haste  by  the  contractor,  and,  therefore, 
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is  usually  pessimistic  in  order  to  cover  estima¬ 
ting  errors.  Nevertheless,  when  the  change  is 

incorporated,  funds  in  the  full  amount  of  the  v 

ceiling  price  must  also  be  obligated  on  the  con¬ 
tract.  From  that  time  until  the  price  of  the 
change  is  determined,  through  negotiation  of  a 
supplemental  agreement,  the  contract  ostensibly 
contains  more  funds  than  are  required  for  the 
tasks  and  the  contractor  is  effectively  without 
an  incentive  to  control  costs  [6:55-56]. 

In  a  study  of  change  order  administration  within  the 
Army  Materiel  Command  (AMC) ,  Williams  and  Beeckler  report  a 
second  factor  to  consider  in  the  use  of  the  change  order. 

This  factor  also  stems  from  the  practice  of  obligating  excess 
funds  in  the  full  amount  of  a  pessimistic  ceiling  price  when 
the  change  order  is  issued.  Williams  and  Beeckler  found  that 
the  use  of  a  change  order  can  unnecessarily  prevent  the  pro¬ 
gram  manager  from  implementing  additional  programs  which  are 
competing  for  the  funds  obligated  by  the  change  order  [15:17]. 

A  third  factor  to  consider  in  the  use  of  a  change 
order  is  a  possible  loss  of  negotiating  leverage.  In  an 
effort  to  retain  cost  control  and  recover  the  use  of  obli¬ 
gated  funds  the  DAR  prescribes  "that  price  adjustments  re¬ 
sulting  from  unpriced  changes.  .  .  shall  be  negotiated  in  the 

shortest  practicable  time  [6:56]."  Williams  and  Beeckler 
point  out,  however,  that  there  is  an  adverse  result  associ¬ 
ated  with  the  requirement  for  timely  def initization  of  change 
orders.  Namely,  that  in  an  effort  to  promptly  negotiate  an 
equitable  adjustment  the  negotiator  may  make  expedient  con¬ 
cessions  which  favor  the  contractor  [15:26]. 
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A  fourth  factor  to  consider  in  the  use  of  the  change 
order  is  the  possibility  that  the  contractor  may  use  the 
change  order  to  "get  well."  This  factor  is  discussed  in 
McMaster's  guide  as  well  as  J.D.  Graham's  report,  Change 
Orders  -  Some  Control  and  Pricing  Aspects.  In  both  sources 
the  authors  display  concern  over  the  likelihood  that  con¬ 
tractors  may  use  the  change  order  to  reap  a  business  or  pro¬ 
fit  "windfall,"  or  to  "get  well"  when  the  basic  contract  is 
in  financial  trouble  due  to  mismanagement  [2:4,  6:56], 

Several  sources  identified  timeliness  as  a  fifth 
factor  to  consider  when  using  the  change  order.  Martin  et 
al  reported  that  in  a  survey  administered  to  approximately 
300  experienced  DoD  procurement  personnel  representing  all 
DoD  procurement  agencies  "all  [agencies]  responded  that  the 
unilateral  change  order  issued  pursuant  to  the  changes 
clause  was  the  most  timely  (contract  modification  instru¬ 
ment)  "  [5:24:110].  Williams  and  Beeckler  similarly  reported 
in  their  study  Change  Order  Administration  that: 

There  are  occasions  when  quick  implementation 
of  the  change  is  necessary  in  order  to  elimi¬ 
nate  a  bottleneck  or  solve  an  engineering 
problem.  Time  may  not  permit  the  negotiation 
of  a  bilateral  agreement.  In  such  situations 
the  changes  clause  gives  the  contracting 
officer  the  right  to  make  a  change  by  means 
of  a  written  change  order  [15:6], 

McMaster  also  discussed  the  timeliness  of  the  change  order 
in  his  report  A  Procurement  Guide  for  Special  Projects  Of¬ 
ficers  where  he  stated  that  in  the  case  of  special  projects: 
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The  need  to  incorporate  the  revisions  becomes 
urgent  lest  the  program  schedule  be  delayed. 

Urgent  changes  in  the  requirements  can  be  in¬ 
corporated  by  a  change  order  prior  to  price 
negotiations  provided  a  ceiling  price  for  the 
changes  has  been  received  by  the  contracting 
officer  from  the  contractor  [6:55]. 

Martin  et  <il  discuss  a  sixth  factor  which  should  be 

considered  when  using  a  change  order.  This  factor  is  the 

timing  of  the  change.  Martin  et  al  assert  that: 

Uncontrolled  changes  often  lead  to  different 
product  configurations  that  seriously  compli¬ 
cate  the  operation  and  support  i  '  the  equipment 
in  the  field.  Poorly  timed  changes  may  dis¬ 
rupt  contractor  operations  unnecessarily, 
create  scrap  or  rework,  and  generally  increase 
cost  to  the  government  [5:3]. 

With  respect  to  the  supplemental  agreement,  the  lit¬ 
erature  discussed  five  factors  to  consider  in  the  use  of  a 
supplemental  agreement.  The  first  three  were  generated  by  a 
survey  administered  to  300  procurement  personnel.  Martin  et 
al  report  that  survey  respondents  from  all  agencies  ranked 
the  supplemental  agreement  as  the  most: 

1.  flexible 

2.  manageable,  and 

3.  cost  effective 

contract  modification  instrument  [5:108]. 

The  fourth  factor  to  consider  in  the  use  of  the  sup¬ 
plemental  agreement  is  DoD  procurement  policy.  The  Defense 
Acquisition  Regulation  (DAR)  requires  the  use  of  the  supple¬ 
mental  agreement  "in  preference  to  a  change  order  when  a 
supplemental  agreement  is  considered  feasible,  even  though 
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authority  exists  to  accomplish  the  modification  by  change 
order  [12:26:7].  Williams  and  Beeckler  explain  the  prefer¬ 
ence  for  the  supplemental  agreement  by  observing  that  with 
the  supplemental  agreement  the  government  and  the  contractor 
can  agree  to  terms  "regarding  the  extent  of  the  change,  the 
price  of  the  change,  and  any  required  adjustment  to  the  de¬ 
livery  schedule"  [15:6]  before  granting  authorization  to 
proceed , 


A  fifth  factor  to  consider  in  the  use  of  a  supplemen¬ 
tal  agreement  is  the  impact  of  the  delayed  "start  date" 
which  is  associated  with  the  supplemental  agreement  [5:110]  . 
In  a  report  to  the  Congress  entitled  Opportunity  to  Reduce 
Costs  and  Improve  Aircraft  Through  Prompt  Processing  of  Engi¬ 
neering  Change  Proposals  the  Comptroller  General  of  the 
United  States  reported  that: 

Usually,  some  aircraft  are  in  production 
while  the  proposed  engineering  change  is  being 
evaluated.  Delays  in  processing  the  change 
proposal  can  increase  the  number  of  unchanged 
aircraft  completed  and  delivered  to  the  opera¬ 
ting  forces.  Once  those  aircraft  are  delivered 
to  the  users,  the  change  could  be  delayed  for 
months  or  years  or  never  be  made  at  all.  More¬ 
over,  making  such  changes  after  production  is 
generally  more  expensive  [11:1]. 

McMaster  takes  a  similar  position. 

In  case  of  added  requirements,  the  price 
of  the  modification  is  the  result  of  the  cost 
of  work  added  plus  the  cost  of  reaccomplishing 
previously  performed  work  --  work  made  useless 
—  in  order  to  integrate  the  new  requirement 
into  the  system.  For  a  given  added  require¬ 
ment  the  magnitude  of  the  work  made  useless 
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will  increase  with  contract  age.  The  cost  of 
the  work  made  useless  constitutes  an  undesir¬ 
able  cost  growth  since  it  represents  lost 
return  of  government  investment  [5:57]. 

Finally,  an  underlying  disadvantage  of  the  delayed  "start 

date"  offered  by  Martin  et  a_l  is  that  it  increases  the  time 

that  the  users  must  live  with  the  deficiency,  indeed  "delayed 

changes.  .  .  may  jeopardize  corrective  actions  designed  to 

avoid  fatal  or  serious  injury  to  operating  personnel  [2:3]. 

Summary 

This  literature  review  has  identified  many  factors 
which  should  be  considered  when  deciding  between  the  change 
order  and  the  supplemental  agreement.  The  literature  has 
revealed  that  with  the  change  order  there  is  a  loss  of  cost 
control  [6:56].  Secondly,  the  use  of  the  change  order  obli¬ 
gates  funds  which  cannot  be  used  elsewhere  until  the  change 
is  definitized  [15:17].  A  third  factor  to  consider  is  that 
the  requirement  for  the  timely  def initization  of  change  or¬ 
ders  can  undermine  the  negotiator's  leverage  and  result  in 
concessions  which  favor  the  contractor  [15:26].  Addition¬ 
ally,  there  is  always  the  possibility  that  the  contractors 
are  using  the  change  order  to  compensate  for  their  "buy  in" 
prices  or  mismanagement  [2:4,  6:56].  Despite  these  limita¬ 
tions  the  change  order  is  the  most  expedient  way  to  incorpor¬ 
ate  a  change  [4:10].  Finally,  the  change  order  requires 
careful  planning  to  insure  the  changes  are  controlled  and  do 
not  result  in  differing  configurations  or  excess  scrap  or 
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rework  [5:3]  . 


For  the  supplemental  agreement,  the  literature  indi¬ 
cated  that  it  is  more  flexible,  manageable  and  cost  effec¬ 
tive  than  the  change  order  [5:108].  Furthermore,  the  DAR 
prescribes  the  use  of  the  supplemental  agreement  whenever 
practicable  [12:26:7];  however,  the  decision  to  use  a  supple¬ 
mental  agreement  must  consider  the  associated  time  delay 
which  can  result  in  a  delay  in  production  effectivity,  poten¬ 
tially  greater  retrofit  costs,  and  periods  of  hazardous  op¬ 
erational  use  [11:1,  6:57,  5:31]. 

In  conclusion,  the  literature  shows  that  authorizing 
the  contractor  to  proceed  prior  to  def initization  of  the 
effort  as  with  change  orders  results  in  a  higher  cost  because 
the  contractor  is  essentially  working  on  a  cost  plus  basis 
and  therefore  has  less  incentive  to  control  costs.  On  the 
otnerhand,  the  literature  has  indicated  that  waiting  to  de- 
finitize  the  effort  prior  to  authorizing  the  contractor  to 
proceed  as  with  supplemental  agreements  results  in  a  more 
costly  retrofit  effort.  A  reasonable  rule  for  determining 
which  method  should  be  used  to  modify  the  contract  where 
safety  factors  do  not  overrule  in  favor  of  the  change  order 
is  to  estimate  the  cost  for  both  alternatives  and  use  the 
least  expensive  one  [8:7], 

Problem 

Program  Managers  must  frequently  decide  whether  to 
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incorporate  a  given  ECP  into  an  existing  contract  by  change 
order  or  supplemental  agreement.  These  managers  are  fully 
aware  of  the  policy  which  favors  the  supplemental  agreement. 
There  has  been  only  limited  quantitative  research  comparing 
the  cost  and  schedule  effects  of  the  change  order  and  the 
supplemental  agreement.  Hence  it  is  useful  to  investigate 
quantitatively,  the  cost  and  schedule  effects  of  these  two 
modification  methods. 

Research  Objectives 

There  are  two  objectives  of  this  project.  The  first 
is  to  quantitatively  compare  the  cost  and  schedule  effects 
of  the  change  order  and  the  supplemental  agreement.  The 
second  is  to  devise  and  test  a  model  to  assist  program  man¬ 
agers  in  determining  whether  an  ECP  should  be  incorporated 
by  a  change  order  or  a  supplemental  agreement. 

Research  Questions 

1.  How  does  the  price  adjustment  factor  for  change 
orders  compare  to  the  price  adjustment  factor  for  supplemen¬ 
tal  agreements  in  those  instances  where  the  program  office 
has  received  an  unpriced  proposal  and  can  select  either 
method  of  contractual  incorporation? 

2.  How  does  the  time  required  to  incorporate  a 
change  order  compare  to  the  time  required  to  incorporate  a 
supplemental  agreement  in  those  instances  where  the  program 
office  has  received  an  unpriced  proposal  and  can  select 
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either  method  of  contractual  incorporation? 


3.  For  those  situations  where  the  contractor  sub¬ 
mits  an  unpriced  ECP,will  the  proposed  model  help  program 
managers  predict  which  method  of  contractual  incorporation 
will  result  in  the  lowest  negotiated  price? 

Scope 

This  research  effort  will  analyze  the  cost  and 
schedule  effect  of  unilateral  change  orders  and  bilateral 
supplemental  agreements  which  are  subject  to  the  changes 
clause.  The  contract  modifications  studies  will  be  re¬ 
stricted  to  those  which  are  generated  by  engineering  change 
proposals . 

Specifically,  this  study  will  focus  on  all  change 
orders  and  supplemental  agreements  issued  against  the  two 
major  production  contracts  used  to  acquire  the  Fairchild 
Republic  Company  A-10A  weapon  system  during  1981. 
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CHAPTER  II 


METHODOLOGY 

Introduction 

This  study  was  conducted  in  two  phases.  Phase  I  de¬ 
termined  the  cost  and  schedule  characteristics  of  the  change 
order  and  supplemental  agreement.  During  Phase  II  the  cost 
and  schedule  characteristics  of  the  change  order  and  supple¬ 
mental  agreement  determined  in  Phase  I  along  with  information 
provided  in  the  ECP  were  used  to  test  a  quantitative  model 
designed  to  predict  the  least  costly  method  of  modifying  the 
contract . 


Population 

The  data  population  for  this  study  consists  of  all 
change  orders  and  supplemental  agreements  issued  pursuant  to 
the  changes  clause  which  incorporated  an  ECP  into  an  A-1Q 
weapon  system  contract. 


Sample 

The  study  sample  space  will  include  only  contract 
modifications  issued  against  contracts  F33657-78-C-0135  and 
F33657-79-C-0502  during  calendar  year  1981.  These  major  pro¬ 
duction  contracts  with  the  Fairchild  Republic  Company  were 
used  to  acquire  the  A-10A  Thunderbolt  II  close  air  support 
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aircraft-  Additionally,  this  study  was  limited  to  cases 
where  the  ECP  was  submitted  as  an  unpriced  effort  with  a  not- 
to-exceed  price. 


Phase  I 


Purpose 

The  first  phase  of  this  research  effort  determined 
the  cost  and  schedule  character istics  of  the  change  order 
and  supplemental  agreement.  Successful  completion  of  Phase 
I  answers  research  questions  one  and  two.  The  steps  required 
to  complete  Phase  I  are  illustrated  in  Figure  1  "Phase  I  - 
Procedures . " 

Procedures 

Step  1:  Gather  data.  The  first  step  in  completing 
Phase  I  was  to  identify  the  discrete  sample  points  which  were 
used  in  this  research  effort.  The  sample  points  for  this 
study  were  extracted  from  the  contract  modification  log 
books  for  contracts  F33657-73-C-0135  and  F33657-79-C-0502 
which  are  located  in  the  Tactical  Systems  Contracting  Office 
in  Aeronautical  Systems  Division.  Once  the  appropriate  con¬ 
tract  modifications  had  been  identified,  the  raw  data  required 
to  perform  both  Phase  I  and  Phase  II  was  extracted  from  the 
appropriate  contract  files  and  ECP  folders.  The  contract 
files  and  the  ECPs  are  also  located  in  Aeronautical  Systems 
Division . 


Step  2:  Sort  data  into  categories.  Once  the  data 
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Gather  Data 


(i) 


Sort  Data 
into 

Categories 


(2) 


Change 

Orders 

-  i 

Supplemental 

Agreements 

1 

Compute 

Price 

Ad j  ustment 

Factors , 

Mean,  and 

(3)  (4) 

\  / 

Compute 

Price 

Adjustment 
Factors , 

Mean,  and 

|  9  51 

C.I . 

95%  C.I. 

Compare 

Price  Adjustment 
of  Change  Order  to 
Supplemental 
Agreement 


Compute 

Compute 

Processing 

Processing 

Times , 

Times , 

Mean  and 

(6) 

(7) 

Mean  and 

95%  C.I . 

RQ2 

951  C.I. 

Compare 

Processing  Time 
of  Change  Order 
to  Supplemental 
Agreement 


V 


(8) 


(#)  corresponds  to 
step  number  in  text 

Fig.  1.  Phase  I  -  Procedures 
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was  collected,  it  was  sorted  into  two  categories:  change 


orders  and  supplemental  agreements.  The  contract  modifica¬ 
tion  form  (Applicable  AFSC  Form  702,  block  14)  was  used  to 
distinguish  between  change  orders  and  supplemental  agree¬ 
ments  . 

Step  3:  Compute  price  adjustment  factors,  mean  and 
95%  confidence  interval  for  change  orders. 

The  price  adjustment  factor  for  change  orders 
(PAF  )  was  calculated  as  follows: 

p  -  p  . 

PAFC  =  — - - 1-  (1) 

Ps 

where : 

Ps  is  the  ceiling  price  (in  dollars)  associated 
with  an  unpriced  ECP.  Pg  is  always  a  not- to - 
exceed  price  in  ECPs  submitted  acainst  the  con¬ 
tracts  used  in  this  study. 

Pf  is  the  defimtized  price  (in  dollars'  of  an 
ECP  as  negotiated  in  a  bilateral  supplemental 
agreement.  ECPs  incorporated  by  change  order 
are  definitized  by  a  supplemental  agreement. 

Once  PAFC  was  determined  for  each  change  order,  a  mean  and 
95%  confidence  interval  were  established. 

Step  4 :  Compute  price  adjustment  factors,  mean,  and 
95%  confidence  interval  for  supplemental  agreements. 


16 


'I 


The  price  adjustment  factors  for  supplemental  agree¬ 
ments  (PAFS)  were  calculated  in  the  same  manner  as  PAFC. 

Once  PAFs  was  determined  for  each  supplemental  agreement, 
a  mean  and  95%  confidence  interval  were  established. 

Step  5;  Compare  price  adjustments  of  change  order 
to  supplemental  agreements. 

Completion  of  Step  5  answered  research  question  1. 
Essentially,  this  step  involved  comparing  the  statistics 
determined  in  Step  3  for  change  orders  to  those  determined 
in  Step  4  for  supplemental  agreements. 

Step  6:  Compute  processing  times,  mean,  and  95% 
confidence  interval  for  change  orders. 

The  processing  times  were  calculated  as  follows: 

Tc  =  Tf  -  Tb  (2) 


where : 

Tc  is  the  number  of  days  between  the  date  the 
program  office  receives  an  ECP  and  the  date  the 
contractor  is  authorized  by  change  order  to  pro¬ 
ceed  with  the  effort. 

Tf  is  an  integer  value  which  corresponds  to  the 
effective  date  of  the  contract  modification  which 
authorizes  the  contractor  to  begin  work. 

Tb  is  an  integer  value  which  corresponds  to  the 
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date  the  program  office  receives  the  ECP . 

Once  Tc  was  determined  for  each  change  order  a  mean  and  95% 
confidence  interval  were  determined. 

Step  7:  Compute  processing  times,  mean  and  95% 
confidence  intervals  for  supplemental  agreements. 

Ts  is  the  number  of  days  between  the  date  the  pro¬ 
gram  office  receives  an  ECP  and  the  effective  date  of  the 
supplemental  agreement  which  authorized  the  contractor  to 
proceed  with  the  effort.  The  processing  time  for  supplemen¬ 
tal  agreements  (T  )  were  calculated  in  the  sane  manner  as 
Tc .  Once  Tg  was  determined  for  each  supplemental  agreement 
the  mean  and  95%  confidence  interval  were  determined. 

Step  8:  Compare  the  processing  time  of  change 
orders  to  supplemental  agreements. 

This  final  step  compared  the  statistics  determined 
in  Step  6  for  change  orders  with  those  determined  for  sup¬ 
plemental  agreements  in  Step  7.  Completion  of  Step  8  an¬ 
swered  research  question  2. 

General  Formulas 

The  means  calculated  in  Steps  3,  4,  6,  and  7  were 
calculated  as  follows: 

n 


n 
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vhere : 


6 ,  and 


where : 


normal , 
known . 

and  8 . 
cedures 


x  =  sample  mean 
n  =  the  number  of  observations 
x^  =  the  i-th  observation 

The  confidence  intervals  calculated  in  Steps  3,  4, 
were  calculated  as  follows  [3:325]: 

x  -  t  — <  u  ^  x  +  t  — § —  (4) 

(et/2,v)  \/n—  (ol/2,v)  /~n 

x  =  sample  mean 

t  =  constant  associated  with  a  level  of  signi¬ 
ficance  and  the  degrees  of  freedom  for  a 
distribution  which  is  assumed  normal 

a  =  1  -  the  desired  confidence  interval 

1  -  .95  =  .05,  in  this  case,  (a/2  =  .025) 

v  =  degrees  of  freedom  (n-1) 

S  =  standard  deviation  of  the  sample 

n  =  the  total  number  of  observations 

U  =  the  population  mean 

This  formula  is  used  when  the  population  is  assumed 
and  the  standard  deviation  for  the  population  is  un- 

Hypothesis  tests  were  conducted  to  complete  Steps  5 
These  tests  were  performed  using  the  following  pro- 
[3:378-379] . 
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Test  on  the  difference  between  two  means  (a  2  and 

2 

C2  unknown  but  assumed  equal) . 

1.  Formulate  null  hypothesis. 

K  :  u  -  u  >0 
o  1  2 

2.  Formulate  alternate  hypothesis. 

H  :  u  -  y  <0 
a  1  2 

3.  Establish  desired  level  of  significance  and 
corresponding  critical  region. 


Cl 


.  05 


Compute  the  test  statistic. 


(*1  “  x2)  -  (t!  -  u 2 ) 


(ni  +  n2  -  2) 


/{n1-l)S12  +  (n^-lTsT^ 
\]  rx1  +  n2  -  2 


n2+n2 

n1xn2 


(5) 


All  of  the  variables  used  in  equation  5  were  defined 
following  equation  4,  with  two  exceptions: 

a2  =  population  variance 

p 

S  =  sample  variance 

5.  Compare  the  test  statistic  to  the  critical 

region  and  draw  conclusions  regarding  hypothe¬ 
sis.  If  the  test  statistic  falls  in  the  criti¬ 
cal  region  the  null  hypothesis  can  be  rejected. 

After  all  eight  steps  of  Phase  I  were  completed,  re- 
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gression  analysis  was  used  to  determine  if  there  was  a  cor¬ 
relation  between  the  price  adjustment  factor  obtained  for 
any  given  ECP  and  the  time  required  to  process  that  ECP. 

As  a  starting  point,  it  must  be  understood  that  regression 
describes  the  nature  of  a  relationship  between  variables. 
Correlation  describes  the  strength  of  this  relationship  [15: 
526] .  When  a  regression  is  performed  it  generates  a  sample 
correlation  coefficient,  r.  This  coefficient  is  also  known 
as  the  Pearson  correlation  coefficient  [16:33].  The  Pear¬ 
son  correlation  coefficient  can  take  on  any  value  between 
-1  and  +1.  A  value  near  +1  indicates  a  very  strong  positive 
correlation,  whereas  a  value  near  -1  indicates  a  very  strong 
negative  correlation.  Values  near  zero  indicate  very  weak 
correlations . 

Each  time  regression  was  performed  in  this  study, 
the  sample  correlation  coefficient  (r)  was  tested  for  statis¬ 
tical  significance  as  follows  [3:528]: 

1.  Formulate  null  hypothesis. 

Hq:  p  =  0  (no  correJ ation) 

2.  Formulate  alternate  hypothesis. 

H  :  p  >  0  (there  is  a  correlation) 

cl 

3.  Determine  desired  level  of  significance  and 
corresponding  critical  region. 
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a  :  .05 


critical  value  will  vary  by  the 
number  of  degrees  of  freedom 


4.  Compute  test  statistic. 


t(n-2) 


r/~  n- 2 
/  1-r2 


(6) 


5.  Determine  conclusion.  If  the  test  statistic  is 
greater  than  the  critical  value  the  null  hypo¬ 
thesis  can  be  rejected,  otherwise  it  cannot  be 
rejected . 


Purpose 


Phase  II 


The  purpose  of  Phase  II  was  to  test  a  model  to  de¬ 
termine  its  usefulness  in  assisting  program  managers  in 
deciding  whether  to  incorporate  an  ECP  by  change  order,  or 
by  supplemental  agreement.  The  model  applies  to  those 
cases  where  the  contractor  has  submitted  an  unpriced  ECP 
with  a  not- to-exceed  price.  The  model  procedures  are  illus¬ 
trated  in  Figure  2  -  "Phase  II  -  Procedures." 
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Fig.  2.  Phase  II  -  Procedures 
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Model  Procedures 


Step  1 :  The  first  step  of  the  model  was  to  collect 
the  information  required  to  apply  the  model.  The  following 
information  was  extracted  from  the  ECP : 

C  is  the  total  recurring  retrofit  costs  of  an 

3. 

ECP  (dollars)  —  DD  Form  1692-3,  Block  48  b  (4). 

C.  is  the  total  recurring  production  costs  of  an 
ECP  (dollars)  --  DD  Form  1692-3,  Block  48  a.  (4). 

N  is  the  number  of  systems  requiring  retrofit  in 
and  ECP  (systems)  —  DD  Form  1692,  Block  20  a. 

N,  is  the  sequence  number  of  the  last  system  to 
be  produced  under  the  existing  contract  (usually 
presented  in  an  attachment,  also  available  in 
the  contract) . 

N  is  the  sequence  number  of  the  first  system  to 
received  the  change  while  in  production.  This 
value  is  also  known  as  the  production  effectivity 
—  DD  Form  1692,  Block  18. 

Ps  is  the  ceiling  price  (or  NTE)  of  the  unpriced 
ECP  (dollars)  —  DD  Form  1692-3,  Block  48  (6). 

Ng  is  the  number  of  systems  receiving  the  modifi¬ 
cation  during  production  for  each  contract 
(usually  presented  in  an  attachment  and  can  be  cal¬ 
culated  using  Nr  and  N  ) . 

f  P 
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The  following  information  was  extracted  from  the  contract: 

R  is  the  monthly  production  rate  of  a  system  (sys¬ 


tems/month)  . 

Finally,  Phase  I  generated  the  following  information: 

PAF  is  the  mean  price  adjustment  factor  for 
change  orders. 

PAF  is  the  mean  price  adjustment  factor  for  sup¬ 
plemental  agreements. 

Tc  is  the  mean  processing  time  for  change  orders 
(days) . 

Ts  is  the  mean  processing  time  for  supplemental 
agreements  (days) . 

Step  2:  Compute  the  expected  cost  if  incorporated 
via  change  order  (CCQ) . 


cco  *  ps(1  -  PAFc>  <7> 

Step  3:  Compute  the  estimated  time  delay  (Tp) . 

Td  =  Ts  -  Tc  (8) 

Step  4:  Compute  the  number  of  systems  affected 
by  the  delay  (N^) . 
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(__£_)  (R)  =  N 

30  1 


Step  5:  Compute  increased  retrofit  cost  (Cr) . 
First  compute  recurring  retrofit  cost  per  unit  (C  ) . 


(9) 


=  C. 


N, 


u 


(10) 


Now  the  increased  retrofit  cost  can  be  calculated. 


Cr  =  (Nd)  (Cu) 


(11) 


Step  6:  Compute  decreased  production  cost  (Cp) . 
First  compute  recurring  production  cost  per  unit  (C  ) . 


Cb 
N  = 


=  C, 


(12) 


Now  the  decreased  production  cost  can  be  calculated. 


Cp  =  (Nd) (Cv) 


(13) 


Step  7 :  Compute  expected  cost  of  supplemental 
agreement  (Csa) .  First  compute  expec  ted  revised  price  of 
the  fully  priced  proposal  (p  ) . 
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Pp  *  Ps  "  Cr  -  cp  <14) 

Now  the  expected  cost  if  incorporated  via  supplemental 
agreement  can  be  calculated. 


csa  =  Pp(1  -  PAFs>  <15) 

Step  8:  Compare  Cco  to  Cca. 

Compare  the  estimated  cost  to  incorporate  the  ECP 
via  change  order  (Cco  from  Step  2)  to  the  estimated  cost  to 
incorporate  the  ECP  via  supplemental  agreement. 

Step  9:  Select  the  least  expensive  alternative, 
unless  overriding  issues  require  the  use  of  a  change  order. 
Test  of  Model 

The  validity  of  a  model  is  a  measure  of  the  extent 
to  which  the  model  predicts  what  actually  occurs.  This 
study  employed  three  methods  of  testing  the  validity  of 
this  model.  The  first  method  was  case  analysis.  With  this 
method  the  model  was  applied  to  two  specific  cases  and  the 
estimated  results  were  compared  to  the  actual  results.  The 
second  method  was  correlation  analysis  where  Pearson  corre¬ 
lation  coefficients  were  calculated  to  determine  if  there 
was  a  significant  correlation  between  estimated  and  actual 
results  for  all  sample  points,  change  orders  only  and  sup¬ 
plemental  agreements  only.  Finally,  algebraic  analysis  was 
applied  to  test  the  validity  of  the  model.  Once  the  model 
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was  reduced  to  its  fundamental  arithmetic  relationships,  in¬ 
ferences  were  drawn  concerning  the  validity  of  the  model. 
Sensitivity  Analysis 

After  the  validity  of  the  model  was  tested,  sensi¬ 
tivity  analysis  was  performed  in  an  effort  to  improve  the 
model.  Regression  analysis  was  performed  to  determine  if 
their  was  a  linear  relationship  between  the  original  not- 
to-exceed  price  and  the  price  adjustment  factor.  Regres¬ 
sion  analysis  was  also  conducted  to  ascertain  if  their  was 
a  linear  relationship  between  the  original  not- to-exceed 
price  and  the  processing  time  for  each  effort.  In  cases 
where  the  regression  analysis  indicated  a  linear  relation¬ 
ship  the  model  was  refined  to  include  that  relationship. 

The  correlation  coefficients  were  tested  for  statistical 
significance  with  the  procedures  discussed  earlier  in  this 
chapter  (ref.  Equation  6). 

Finally,  an  analysis  of  variance  (ANOVA)  was  per¬ 
formed  to  determine  if  the  price  adjustment  factors  varied 
as  a  function  of  time.  All  regressions,  correlation  coef¬ 
ficients,  and  the  ANOVA  were  calculated  through  the  use  of 
the  Statistical  Package  for  the  Social  Sciences  (SPSS9)  . 
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CHAPTER  III 


FINDINGS  AND  ANALYSIS 

Introduction 

The  purpose  of  Chapter  III  is  to  present  the  find¬ 
ings  of  this  study  and  apply  those  findings  to  the  acquisi¬ 
tion  environment.  The  chapter  will  first  present  the  find¬ 
ings  of  Phase  I.  The  analysis  of  these  results  will  answer 
research  questions  1  and  2.  The  next  section,  titled 
"Phase  II,"  will  include  two  case  analyses,  correlation 
analysis,  algebraic  analysis,  sensitivity  analysis  and 
model  refinement.  The  analysis  in  Phase  II  will  answer 
research  question  3.  The  application  section  will  then  dis¬ 
cuss  the  use  of  the  findings  from  Phase  I  and  Phase  II  in 
the  acquisition  environment.  The  main  points  of  the  chapter 
will  be  summarized  in  the  final  section. 


Phase  I 


Data 

The  data  search  for  this  study  identified  51  poten¬ 
tial  data  points  within  the  sample  space.  These  data  points 
are  displayed  in  Appendix  A  "Sample  Points."  The  51  poten¬ 
tial  data  points  were  then  sorted  into  two  categories: 

1.  Change  Orders  (Appendix  B  "Change  Orders"). 

2.  Supplemental  Agreements  which  incorporated  ECPs 
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submitted  originally  as  an  unpriced  proposal 
(Appendix  C  "Supplemental  Agreements"). 

Originally,  each  supplemental  agreement  presented 
in  Appendix  C  "Supplemental  Agreements"  would  have  served 
as  an  independent  sample  point.  As  the  data  were  collected, 
however,  it  became  apparent  that  the  only  way  to  compare  a 
not-to-exceed  price  to  a  final  definitized  price  which 
addressed  the  same  number  of  aircraft  was  to  group  the  con¬ 
tract  modifications  which  incorporate  the  same  ECP .  This 
grouping  method  will  adjust  for  the  shifting  effectivity 
which  accompanies  a  delay  in  contractual  incorporation. 
Research  Question  1 

How  does  the  price  adjustment  factor  for  change  or¬ 
ders  compare  to  the  price  adjustment  factor  for  supplemental 
agreements  in  those  instances  where  the  program  office  has 
received  an  unpriced  proposal  and  can  select  either  method 
of  contractual  incorporation?  The  mean  price  adjustment 
factor  for  change  orders  (PAFC)  was  29.432%.  This  means 
that  the  initial  not-to-exceed  price  for  change  orders  was 
reduced  by  an  average  29.432%  as  a  result  of  the  definiti- 
zation  process.  The  95%  confidence  interval  for  the  price 
adjustment  factor  for  change  orders  is  as  follows: 

23.01  <  u  <  35.85 
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This  confidence  interval  indicates  that  PAFC  will  fall  be¬ 
tween  23.01  and  35.85%  95  percent  of  the  time. 

The  mean  price  adjustment  factor  for  supplemental 
agreements  (PAFg)  was  computed  to  be  23.387%,  indicating 
that  for  supplemental  agreements  the  initial  not-to-exceed 
price  was  reduced  an  average  of  23.387%  as  a  result  of  the 
def initization  process.  The  95%  confidence  interval  for 
PAFS  is : 


16.34  <  u  <  30.23 

Figure  3  illustrates  the  distributions  of  price  ad¬ 
justment  factors  for  both  change  orders  and  supplemental 

agreements.  _  _ 

PAFs  PAFC 
23.287  29.432 


Fig.  3.  Distributions  of  Price  Adjustment  Factors 

It  is  clear  from  Figure  3  that  the  mean  price  adjustment 
factor  for  change  orders  is  greater  than  the  mean  price  ad- 
ments  factor  for  supplemental  agreements.  Since  the  price 
adjustment  factor  represents  the  decrease  in  the  price  of 
the  effort,  the  larger  price  adjustment  factor  represents 
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the  least  costly  alternative.  The  overlap  between  the  two 
distributions  indicates  that  it  is  possible  for  a  random 


PAFg  to  be  greater  than  a  random  PAFC.  In  this  situation, 
the  supplemental  agreement  would  be  less  costly  than  a 
change  order;  however,  Figure  3  indicates  this  will  happen 
only  occasionally . 

A  hypothesis  test  was  performed  to  determine  if  the 
mean  price  adjustment  factor  for  change  orders  was  greater 
than  for  supplemental  agreements  at  the  .05  level  of  signi¬ 
ficance.  This  test  indicated  that  it  was  not  statistically 
greater  at  the  .05  level  of  significance  (Appendix  D  "Hypo¬ 
thesis  Tests"  Test  1) .  The  test  was  performed  again  at  the 
.25  level  of  significance.  This  second  test  indicated  that 
the  PAFC  is  statistically  greater  than  PAFS  (Appendix  D 
"Hypothesis  Tests"  Test  2)  . 

The  raw  data  used  to  compute  PAFC  and  PAFS  are  pre¬ 
sented  in  Appendix  E  "Price  Adjustment  -  Change  Orders," 
and  Appendix  F  "Price  Adjustment  -  Supplemental  Agreements" 
respectively . 

Research  Question  2 

How  does  the  time  required  to  incorporate  a  change 
order  compare  to  the  time  required  to  incorporate  a  supple¬ 
mental  agreement  in  those  instances  where  the  program 
office  has  received  an  unpriced  proposal  and  can  select 
either  method  of  contractual  incorporation? 


The  time  required  to  incorporate  a  modification  is 
the  number  of  days  between  the  date  the  program  office  re¬ 
ceives  the  ECP  and  the  date  the  contractor  is  given  authori¬ 
zation  to  proceed.  For  change  orders,  this  time  generally 
includes  the  time  it  takes  the  program  office  to  review  the 
ECP,  make  any  necessary  revisions,  and  issue  the  contract 
modification  document.  For  supplemental  agreements  this 
time  includes  the  time  it  takes  the  contractor  to  fully 
price  the  proposal  and  negotiate  a  settlement  in  addition 
to  the  initial  review  and  revision  process  which  takes 
place.  Consequently,  the  supplemental  agreement  logically 
should  be  the  slower  of  the  two  alternatives.  The  findings 
of  this  study  confirm  this  assertion. 

The  mean  processing  time  for  change  orders  (Tc)  was 
43.05  days.  The  95%  confidence  interval  was: 

27.38  <  u  <  58.21 

The  mean  processing  time  for  supplemental  agreements  was 
186.11  days  with  a  95%  confidence  interval  as  follows: 

114.5  <  m  <  257.72 

Figure  4  illustrates  the  distributions  of  processing 
time  for  both  the  change  order  and  the  supplemental  agree¬ 
ments  . 
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Change  Supplemental  Agreement  (t  ) 

Order  (Tc) 


Fig.  4.  Distributions  of  Processing  Times 

Clearly,  the  change  order  is  the  more  expedient 
method.  A  hypothesis  test  for  the  difference  of  means  con¬ 
firmed  that  T  is  less  than  T_  at  the  .05  level  of  signifi- 

c  s 

cance  (Appendix  D  "Hypothesis  Tests"  Test  3) . 

The  raw  data  used  to  compute  TQ  and  Ts  are  presented 
in  Appendix  G  "Processing  Time  -  Change  Orders,"  and  Appen¬ 
dix  H  "Processing  Time  -  Supplemental  Agreements,"  respec¬ 
tively. 

Additional  Analysis 

The  primary  emphasis  of  Phase  I  has  been  to  deter¬ 
mine  the  nature  of  the  change  order  and  the  supplemental 
agreement  with  respect  to  the  price  adjustment  realized  dur¬ 
ing  def initization  and  processing  time.  This  analysis  can 
be  taken  one  step  further  by  determining  if  there  is  a  cor¬ 
relation  betwc  m  the  price  adjustment  factor  obtained  for 
any  given  ECP  and  the  time  required  to  process  that  ECP. 

Three  regressions  were  performed  as  indicated  in 
Table  1  "Regression  -  PAF  vs  Processing  Time." 
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TABLE  1 


REGRESSION  -  PAF  with  PROCESSING  TIME 


Dependent 

Regression  Variable 

Independent 

Variable 

Sample 

Points 

Pearson 
Correlation 
Coefficient  (r) 

1 

Price 

Adjustment 

Factor 

Processing 

Time 

All 

(1-37) 

.  0082 

2 

PAF 

c 

T 

c 

Change 

Orders 

(1-28) 

.  0907 

3 

PAFS 

T 

s 

Supple¬ 

mental 

Agree¬ 

ments 

(29-37) 

.6426 

The  correlation  coefficients  generated  for  each  regression 
were  tested  for  statistical  significance,  a  =  . 05  (Appendix 
D  "Hypothesis  Tests,  Tests  4,  5,  &  6"). 

The  hypothesis  tests  generated  the  following  conclu¬ 
sions.  For  all  sample  points,  and  for  change  orders  only, 
the  null  hypotheses  could  not  be  rejected  at  the  .05  level 
of  significance.  However,  the  tests  performed  for  regres¬ 
sion  3  indicate  that  there  is  a  positive  correlation  between 
the  price  adjustment  factor  and  the  processing  time  for  sup¬ 
plemental  agreements. 

Remember  that  the  processing  time  for  change  orders 
does  not  include  the  time  required  to  aefinitize  the  effort 
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because  the  contractor  is  given  authorization  to  proceed 
before  the  effort  is  definitized.  Therefore  one  would  not 
expect  the  price  adjustment  factor  to  be  correlated  with 
the  processing  time.  On  the  other  hand,  the  processing 
time  for  supplemental  agreements  does  include  the  time  re¬ 
quired  to  definitize  the  effort.  Consequently,  it  is 
reasonable  to  assume  the  price  adjustment  factor  would  be 
correlated  to  the  time  required  to  process  the  ECP .  The 
hypothesis  tests  confirm  these  assumptions.  Furthermore, 
the  positive  correlation  between  the  price  adjustment  factor 
and  the  processing  time  for  supplemental  agreements  suggests 
that  the  government  obtains  a  more  favorable  settlement  in 
those  cases  where  the  def initization  takes  longer. 


Phase  II 

Case  Analysis 

A  common  method  of  testing  the  validity  of  a  model 
is  case  analysis.  In  case  analysis  the  model  is  applied  to 
actual  situations.  Then  the  model's  estimates  are  compared 
to  the  actual  results  to  determine  if  the  model  validly  pre¬ 
dicts  what  actually  happens.  Two  cases  have  been  selected 
to  test  the  model  developed  in  this  study. 

The  first  case  is  engineering  change  proposal  F2780 
titled  "Eliminate  Chafing  of  Hydraulic  Hoses  and  Electrical 
Wire  in  the  Wings  and  Nacelles."  This  change  was  precipi¬ 
tated  when  an  accident  investigation 
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revealed  that  [an]  inflight  fire  resulted  from 
the  chafing  of  a  left  wing  slat  hydraulic  hose 
against  an  adjacent  electrical  harness,  which 
resulted  in  shorting  of  the  wire  and  localized 
failure  of  the  hose  [1:1]  . 

Fairchild  Republic  Company  proposed  a  fix  which  in¬ 
volved  wrapping  teflon  spiral  wrap  chafe  sleeves  at  the 
affected  areas  and  replacing  existing  hoses  with  hoses 
which  incorporate  a  teflon  tubular  chafe  sleeve. 

The  second  case  is  engineering  change  proposal  F2768 
Rl  titled  "Elimination  of  Water  Intrusion  Problems."  This 
change  was  proposed  in  response  to  reports  of  ground  aborts 
and  electrical/avionic  equipment  problems  resulting  from 
water  intrusion.  The  urgency  of  the  change  was  tied  to  the 
scheduled  NATO  deployment  of  A-10A  aircraft.  The  ECP  pro¬ 
posed  to  install  shields,  grommets,  and  sealant  where  appro¬ 
priate  . 

The  methodology  described  in  Chapter  II  was  applied 
to  the  data  listed  in  Appendix  I  -  "Case  Analysis  Data,"  and 
generated  the  following  results.  For  Case  I  the  model  esti¬ 
mated  that  the  definitized  price  of  the  effort  would  be 
$189,404  if  incorporated  by  change  order,  and  $210,028  if 
incorporated  by  supplemental  agreement.  The  actual  defini¬ 
tized  price  of  the  supplemental  agreement  which  incorporated 
this  ECP  was  $173,334.  In  this  case  the  model  overestimated 
the  price  of  the  supplemental  agreement  by  over  $36,000. 

The  analysis  for  Case  II  generates  similar  results. 
For  Case  II  the  estimated  definitized  price  of  a  change 
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order  was  $346,842.  The  estimated  price  of  a  supplemental 
agreement  was  $382,232.  Again,  the  program  office  selected 
the  supplemental  agreement.  The  actual  definitized  price 
(adjusted  to  estimate  the  price  of  alternative  II  and  III 
only)  was  $351,174.  As  before,  the  model  overestimated  the 
definitized  price  of  the  supplemental  agreement,  this  time 
by  over  $31,000. 

The  results  of  these  two  analyses  suggest  that  the 
model  overestimates  the  price  of  the  supplemental  agreement. 
This  is  not  necessarily  the  case  however.  On  the  contrary, 
the  model  underestimated  the  price  to  incorporate  an  ECP  by 
supplemental  agreement  in  five  of  the  eight  cases  used  in 
this  study.  This  discrepancy  reveals  a  major  flaw  with 
validation  by  case  analysis.  Case  analysis  often  forces  the 
analyst  to  draw  conclusions  on  too  limited  a  sample.  A 
second  method  which  can  be  used  to  test  the  validity  of  this 
model  is  correlation  analysis.  Correlation  analysis 
accounts  for  all  of  the  cases  in  the  sample,  thereby  over¬ 
coming  the  primary  deficiency  of  case  analysis. 

Correlation  Analysis 

Correlation  analysis  can  be  used  to  see  if  the  model 
is  making  accurate  predictions  by  determining  if  there  is  a 
significant  correlation  between  the  actual  results  and  the 
estimated  results.  The  model  was  applied  to  all  37  sample 
points  to  generate  the  estimate  which  corresponds  to  the 
decision  of  the  program  office.  For  example,  if  the  program 
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office  incorporated  the  effort  by  change  order,  the  estima¬ 
ted  price  of  a  change  order  (Cco)  was  calculated.  The 
actual  prices  were  extracted  from  the  contract  modif  icatior.s . 
Appendix  J  "Correlation  Analysis"  contains  the  data  used  for 
this  correlation  analysis.  Pearson  correlation  coeffici¬ 
ents  (r)  were  generated  for  three  sets  of  sample  points: 
all  contract  modifications,  change  orders  only,  and  supple¬ 
mental  agreements  only.  The  Pearson  correlation  coeffici¬ 
ents  for  these  three  sets  of  sample  points  were  .9986, 

.9986,  and  .9812  respectively.  Hypothesis  tests  for  the 
significance  of  correlation  were  conducted  as  described  by 
equation  6  in  Chapter  II  (Appendix  D  -  "Hypothesis  Tests," 
test  7,8,  and  9).  These  hypotheses  indicate  that  in  all 
three  cases  there  is  a  positive  correlation  which  is  statis¬ 
tically  significant  at  the  a  =  .05  level  of  significance. 

In  fact,  the  correlation  is  even  significant  when  a  =  .0005 
[3:A-49].  While  the  hypothesis  tests  indicate  a  very  strong 
correlation  between  the  estimated  and  actual  results,  this 
correlation  cannot  be  attributed  solely  to  the  validity 
of  the  model. 

It  must  be  remembered  that  the  actual  and  estimated 
values  are  both  directly  proportional  to  the  original  not- 
to-exceed  prices  quoted  for  each  effort.  Consequently,  a 
part  of  the  correlation  is  due  to  the  manner  in  which  the 
actual  and  estimated  results  were  generated,  and  not  the 
validity  of  the  model. 
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Algebraic  Analysis 

The  validity  of  the  model  can  also  be  tested  alge¬ 
braically.  Essentially,  the  model  reduces  to  the  following: 

P  x  .70568  =  C  (16) 

S  CO 

and 

(P  ±  adjustment  for  delay)  x  .76613  =  Csa  (17) 

where  .70568  and  .76613  are  equal  to  one  minus  the  price  ad¬ 
justment  factors  generated  in  Phase  I  of  this  study.  The 
break  even  point  is  that  point  at  which  the  price  of  a  change 
order  (Cco)  is  equal  to  the  price  of  a  supplemental  agree¬ 
ment  (Csa) .  This  can  be  determined  by  setting  the  equation 
for  Cco  equal  to  the  equation  for  Csa  or: 

.  70568  (Ps)  =  . 76613  (Ps)  +  (Cr  -  Cp)  (18) 

where : 

Ps  =  not-to-exceed  price 

Cr  =  price  increase  due  to  increased  retrofit 
effort 

CD  =  price  decrease  due  to  decreased  production 
effort 

simplifying : 


Cp  -  Cr  =  (.76613  -  .70568)  (Ps)  (19) 
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Cp  =  .06045  Ps  +  cr  (20) 

The  model  was  designed  to  balance  the  loss  of  cost 
control  associated  with  the  change  order,  with  the  more 
costly  retrofit  effort  associated  with  the  supplemental 
agreement.  This  balance  would  have  been  achieved  if  the 
mean  price  adjustment  factor  for  supplemental  agreements, 
exceeded  the  mean  price  adjustment  factor  for  change  orders. 
It  did  not,  consequently,  the  model  will  almost  always  rec¬ 
ommend  the  use  of  the  change  order.  Equation  20  confirms 
this  assertion.  Remember  that  the  cost  of  retrofit  is 
generally  more  expensive  than  the  cost  of  making  the  change 
in  production  [11:1].  Equation  20  indicates  that  the  break 
even  point  occurs  when  C  is  greater  than  Cr.  Since  Cr  is 
generally  greater  than  Cp  [11:1]  the  model  will  almost 
always  recommend  the  use  of  the  change  order. 

This  conclusion  does  not  necessarily  indicate  that 
the  model  is  invalid.  Rather  it  suggests  that  either: 

1.  the  model  has  underestimated  the  price  adjust¬ 
ment  factor  for  supplemental  agreements,  and 
unduly  favors  the  change  order,  or 

2.  the  assumption  that  a  loss  of  cost  control  makes 
the  change  order  more  expensive  is  overstated. 

Sensitivity  Analysis 

Chapter  II  describes  several  variables  which  are 
used  as  inputs  to  the  model.  Most  of  the  values  for  these 
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variables  are  extracted  directly  from  the  ECP .  However, 
four  variables  must  be  estimated  by  the  program  manager. 
These  variables  are: 

1.  The  price  adjustment  factor  for  change  orders, 

2.  The  price  adjustment  factor  for  supplemental 
agreements , 

3.  The  processing  time  for  change  orders,  and 

4.  The  processing  time  for  supplemental  agree¬ 
ments  . 

Thus  far,  the  analysis  has  used  the  values  of  PAFC , 
PAFS,  Tc,  and  Ts  generated  in  Phase  I  as  estimates  for  the 
four  variables  described  above.  Sensitivity  analysis  pro¬ 
vides  the  opportunity  to  refine  the  model  by  using  regres¬ 
sion  to  improve  these  estimates. 

First,  regression  analysis  was  applied  to  deter¬ 
mine  if  there  was  a  linear  relationship  between  the  price 
adjustment  factors  and  the  initial  not-to-exceed  price 
submitted  with  the  ECP.  If  there  was  a  relationship,  the 
estimates  for  the  price  adjustment  factors  could  be  made 
sensitive  to  the  initial  not-to-exceed  price.  Three  re¬ 
gressions  were  performed  as  indicated  in  Table  2  "Regres¬ 
sion  -  PAF  with  Ps."  Each  Pearson  correlation  coeffici¬ 
ent  generated  was  tested  for  statistical  significance  in 
Appendix  D  "Hypothesis  Tests,"  Tests  10-12.  In  all  three 
cases,  the  relationship  between  the  price  adjustment  factor 
and  the  not-to-exceed  price  was  not  statistically 
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TABLE  2 


REGRESSION  -  PAF  WITH  Ps 


REGRESSION 

DEPENDENT 

VARIABLE 

INDEPENDENT 

VARIABLE 

SAMPLE 

POINTS 

PEARSON 

CORRELATION 

COEFFICIENT 

1 

PAF  c ,  PAFS 

ps 

all 

.  0349 

2 

PAFC 

ps 

Change 

Orders 

.  0731 

3 

PAFS 

Ps 

Supple¬ 

mental 

Agree¬ 

ments 

.4997 

significant  at  the  .05  level  of  significance.  Consequently 
a  variable  price  adjustment  factor  would  not  improve  the 
model . 

Sensitivity  analysis  was  also  applied  to  improve  the 
estimated  time  delay  (Ts  -  Tc) .  Regression  analysis  was 
used  to  investigate  the  existence  of  a  linear  relationship 
between  processing  time  and  the  not- to-exceed  price  (Ps)  as 
indicated  in  Table  3  "Regression  -  T  with  P  The  hypo¬ 

thesis  test  for  the  statistical  significance  of  the  Pearson 
correlation  coefficient  for  regression  1  indicated  that 
there  was  not  a  statistically  significant  correlation  between 
processing  time  and  the  initial  not-to-exceed  price  for  all 
sample  points  (ref:  Appendix  D:  Hypothesis  Tests"  test  13). 

However,  the  hypothesis  tests  conducted  for  regres- 
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TABLE  3 


REGRESSION  -  T  WITH  Pg 


REGRESSION 

DEPENDENT 

VARIABLE 

INDEPENDENT 

VARIABLE 

SAMPLE 

POINTS 

PEARSON 

CORRELATION 

COEFFICIENT 

1 

Tc'  Ts 

PS 

all 

.  0630 

2 

Tc 

P 

s 

Change 

Orders 

.  4776 

3 

Ts 

ps 

Supple¬ 

mental 

Agree¬ 

ments 

.7537 

sions  2  and  3  indicated  that  there  was  a  statistically  sig¬ 
nificant  linear  relationship  for  both  change  orders  and  sup¬ 
plemental  agreements  (ref:  Appendix  D  "Hypothesis  Tests", 
test  14  and  15) .  The  regression  performed  indicates  that 
for  change  orders: 

y  =  .0000062  (x)  +  34.3226 

processing 

time  (Tc)  =  .0000062  (initial  price)  +  34.3226 

days 

and  for  supplemental  agreements: 

y  =  .0003004  (x)  +  118.155 

processing 

time  (Ts)  =  .0003004  (initial  price)  +  118.155 

days 

These  functions  can  be  used  to  refine  the  model. 

The  delay  can  be  expressed  as  the  difference  between  the 
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processing  time  for  each  alternative,  or: 

delay  =  (.0003004x  +  118.155)  -  (.0000062x  +  34.3446) 

(21) 

Since  x  represents  the  same  initial  price  in  both  expressions  , 
this  equation  simplifies  to  the  following: 

delay  =  .0002942x  +  83.83  (22) 

Previously,  the  delay  was  calculated  to  be  143  days  in  all 
cases.  Now  the  delay  will  be  calculated  per  equation  22. 

For  instance,  if  an  ECP  had  a  not-to-exceed  price  of  $560, 

000,  the  model  would  now  predict  a  delay  of  248  days. 

Correlation  analysis  was  performed  to  determine  if 
the  refinement  was  in  fact  an  improvement.  The  estimated 
price  for  each  supplemental  agreement  was  recalculated  using 
this  delay  refinement.  The  revised  estimates  were  then  re¬ 
gressed  with  the  actual  results.  This  regression  generated 
a  Pearson  correlation  coefficient  (r)  of  .9928.  The  corre¬ 
lation  between  the  estimated  and  actual  price  of  supplemen¬ 
tal  agreements  prior  to  the  refinement  was  .9812.  Since  the 
new  correlation  coefficient  was  higher,  the  model  was  re¬ 
vised  to  include  the  "delay  refinement."  Step  3  of  the 
model  is  changed  to  the  following: 

Step  3:  Compute  the  estimated  time  delay  (t^) . 
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Td  =  . 0002942  (P  )  +  83.83 

Application 

This  model  can  be  applied  to  another  program.  The 
relationships  built  into  the  model  are  well  substantiated  in 
the  literature  as  well  as  actual  practice.  The  success  of 
the  model  depends  on  how  accurately  the  using  program  office 
estimates : 

1.  the  price  adjustment  factor  for  change  orders 
(PAFc) 

2.  the  price  adjustment  factor  for  supplemental 
agreements  (PAFs) 

3.  the  time  delay  associated  with  the  supple¬ 
mental  agreement  (TD) . 

In  this  study  the  price  adjustment  factor  for  change 
orders  (PAFC)  was  estimated  by  taking  a  sample  of  28  chance 
orders  executed  during  a  single  year  and  calculating  a  mean 
(PAFC) . 

The  95%  confidence  interval  was  relatively  tight, 
ranging  from  approximately  23%  to  36%  so  the  estimate  of 
29.432%  is  a  fairly  reliable  estimate  for  this  study.  This 
rate  should  not  however,  be  used  by  a  different  program 
office  because  this  rate  is  a  function  of  several  factors 
which  are  specific  to  the  program  used  in  this  study. 

There  are  several  factors  which  the  program  manager  should 
consider  when  formulating  an  estimate  of  the  price  adjustment 
factor  for  change  orders. 

1.  What  is  the  risk  factor  applied  by  the  contractor 
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when  submitting  a  not- to-exceed  price?  Some  contractors 
will  add  a  risk  factor  of  20%  while  others  may  double  their 
estimates  of  what  the  ECP  should  cost.  The  risk  factor 
applied  by  the  contractor  can  be  obtained  through  conversa¬ 
tions  with  the  contractor's  pricing  staff.  Oftentimes  it 
is  provided  in  the  ECP.  The  program  office  contracting 
staff  will  also  have  a  good  idea  of  how  large  a  risk  factor 
is  applied. 

2.  What  phase  of  the  acquisition  process  is  the 
program  in?  Contractors  will  generally  apply  a  larger 
risk  factor  early  in  the  development  phase  when  the  system 
design  is  still  evolving.  Once  the  program  office  begins 
production  however,  the  risk  factor  stabilizes.  This  asser¬ 
tion  was  confirmed  through  an  analysis  of  variance  (ANOVA) . 
The  ANOVA  was  applied  to  determine  if  price  adjustment  fac¬ 
tors  calculated  early  in  the  production  phase  could  be  used 
to  predict  price  adjustment  factors  later  in  production. 

The  data  for  this  study  were  divided  into  8  time  treatment 
groups.  The  ANOVA  indicated  the  price  adjustment  factors 
did  not  change  as  the  system  progressed  to  the  later  stages 
of  the  production  effort. 

3.  What  is  the  nature  of  the  environment  in  the 
using  programs  contracts  office?  If  the  office  is  over¬ 
worked  and  unable  to  provide  sufficient  support  to  rigorous 
fact-finding,  or  if  the  pressure  to  definitize  efforts 
quickly  is  undermining  the  negotiators'  leverage,  a  contract 
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office  may  not  be  realizing  an  optimal  price  adjustment. 

These  three  factors  can  be  used  to  formulate  a 
theoretical  estimate  of  the  price  adjustment  factor  for 
change  orders.  This  theoretical  estimate  can  be  formulated 
at  any  point  in  the  acquisition  process.  A  program  office 
may  prefer  to  calculate  an  estimate  based  on  a  sample  mean 
as  in  this  study.  Bearing  in  mind  the  factors  described 
above,  a  mean  can  be  calculated  based  on  any  sample  size. 

A  program  manager  should  be  able  to  increase  his  or  her  con¬ 
fidence  in  the  estimate  by  increasing  the  sample  size.  A 
sample  size  of  30  is  typically  associated  with  the  minimum 
number  of  sample  points  required  to  assume  that  a  distribu¬ 
tion  is  normal. 

This  study  estimated  the  price  adjustment  factor  for 
supplemental  agreements  (PAF  )  in  the  same  manner  that  PAF 

O  V-* 

was  estimated.  A  sample  of  8  supplemental  agreements  exe¬ 
cuted  during  1981  was  used  to  calculate  a  mean  and  95%  con¬ 
fidence  interval.  The  small  sample  size  did  impact  the 
width  of  the  confidence  interval.  As  before,  the  23.387% 
value  of  PAFs  should  not  be  used  by  another  program  office. 

A  theoretical  PAFg  can  be  formulated  by  considering  the  fol¬ 
lowing  factor  in  addition  to  the  three  factors  discussed 
for  change  orders. 

1.  What  is  the  nature  of  the  system?  Aircraft  sys¬ 
tems  typically  involve  retrofit,  which  increases  the  cost 
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of  the  supplemental  agreement.  Different  types  of  systems 
may  not  require  retrofit. 

An  estimate  can  also  be  obtained  statistically  using 
the  procedures  employed  in  Phase  I  of  this  study.  A  larger 
sample  size  will  increase  the  confidence  associated  with  an 
estimate . 

The  third  parameter  which  m  st  be  estimated  is  the 
time  delay  associated  with  the  supplemental  agreement.  This 
study  estimated  this  delay  be  computing  the  difference  be¬ 
tween  the  mean  processing  time  for  change  orders  (T  )  and 
the  mean  processing  time  for  supplemental  agreements  (T  ) . 
This  study  then  refined  this  estimate  by  regressing  process¬ 
ing  time  with  the  initial  not-to-exceed  price.  Another  pro¬ 
gram  office  can  employ  either  of  these  techniques.  Again, 
a  larger  sample  size  will  improve  the  confidence  associated 
with  these  estimates.  A  theoretical  estimate  can  also  be 
formulated,  and  should  consider  the  following  factors: 

1.  What  activities  contribute  to  the  delay?  How 
long  do  these  activities  take?  If  an  ECP  is  deferred  to  al¬ 
low  incorporation  by  supplemental  agreement,  the  ECP  must  be 
fully  priced,  then  the  ECP  must  be  reviewed  again.  Once  the 
change  is  approved  it  must  be  negotiated.  The  contractor  is 
responsible  for  pricing  the  ECP  and  is  probably  the  best 
source  for  estimating  other  time  required  to  complete  that 
activity.  The  program  manager  is  best  qualified  to  estimate 
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how  long  the  program  office  will  review  the  ECP  before  ap¬ 
proving  it.  Finally,  the  contracts  office  is  a  good  source 
to  estimate  the  time  required  to  negotiate  the  effort. 

2.  How  complex  is  the  task?  A  simple  task  is  more 
easily  priced,  reviewed,  and  negotiated  than  a  complex  task. 

3.  How  expensive  is  the  ECP?  The  review  cycle,  and 
the  negotiation  cycle  become  more  rigorous  as  the  cost  of  an 
ECP  increases  above  certain  legal  thresholds. 

4.  How  does  the  contractor  view  incorporation  of 
the  effort?  It  is  possible  that  a  contractor  is  not  in 
favor  of  the  proposed  modification.  This  may  occur  in  cases 
where  the  contractor  feels  that  changes  that  result  from 
changes  made  by  an  associate  contractor  or  made  in  govern¬ 
ment  furnished  equipment  may  adversely  impact  the  system. 

If  the  contractor  is  not  in  favor  of  incorporating  a  pro¬ 
posed  modification  there  is  a  good  chance  it  will  take 
longer . 

5.  How  well  staffed  is  the  contracting  office?  If 
the  office  is  understaffed  the  contract  modifications  requir¬ 
ing  def initization  may  have  to  wait  to  be  negotiated. 

This  section  has  described  how  another  program  of¬ 
fice  should  use  the  model  developed  in  this  study.  The  sec¬ 
tion  has  emphasized  that  the  success  of  the  model  is  heavily 
dependent  on  the  accuracy  of  the  estimates  for  PA?c,  PAFS 
and  TD>  Several  factors  have  been  listed  to  assist  program 
managers  in  developing  accurate  estimates  for  those  parameters. 
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Summary 


Several  objectives  were  accomplished  in  Phase  I. 
First,  the  mean  price  adjustment  factors  for  change  orders 
and  supplemental  agreements  were  calculated  and  compared. 
Second,  the  mean  processing  times  for  change  orders  and  sup¬ 
plemental  agreements  were  calculated  and  compared.  Third, 
correlation  analysis  determined  that  there  was  a  correlation 
between  processing  time  and  the  price  adjustment  factor  for 
supplemental  agreements.  In  Phase  II,  the  model  was  tested 
for  validity  using  case  analysis,  correlation  analysis,  and 
algebraic  analysis.  Sensitivity  analysis  was  then  applied 
to  refine  the  model.  This  sensitivity  analysis  determined 
that  there  was  a  significant  correlation  between  the  initial 
ceiling  price  and  the  processing  time  for  both  change  orders 
and  supplemental  agreements.  These  relationships  were  then 
used  to  refine  the  model.  Finally,  the  application  of  the 
model  was  discussed.  Emphasis  was  placed  on  the  importance 
of  the  estimates  for  PAFC,  PAFS  and  TD-  Several  factors 
were  discussed  to  assist  the  program  manager  make  these 
estimates . 
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CHAPTER  IV 


CONCLUSION 

Summary 

The  two  objectives  of  this  study  were  to  first, 
quantitatively  compare  the  cost  and  schedule  impact  of  the 
change  order  and  the  supplemental  agreement,  and  second, 
to  test  a  model  to  assist  program  managers  in  determining 
whether  an  ECP  should  be  incorporated  by  a  change  order  or 
a  supplemental  agreement. 

Phase  I  of  this  study  determined  that  the  program 
office  could  expect  to  decrease  the  not-to-exceed  price  sub¬ 
mitted  with  an  ECP  by  a  larger  amount  if  the  effort  was  in¬ 
corporated  by  change  order.  In  the  case  of  the  change  order, 
the  not-to-exceed  price  was  decreased  by  an  average  of  29% 
by  the  time  the  effort  was  finally  definitized.  The  mean 
decrease  in  the  price  experienced  in  those  cases  where  the 
program  office  chose  to  incorporate  the  ECP  by  supplemental 
agreement  was  23%.  The  difference  in  mean  price  adjustment 
factors,  however,  was  not  statistically  significant  (a  =  .05). 

Phase  I  also  investigated  the  schedule  impact  of 
delaying  incorporation  of  an  urgent  ECP  until  the  contractor 
could  provide  a  fully  priced  proposal.  This  study  indicates 
that  when  an  ECP  is  incorporated  by  change  order  it  takes 
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the  program  office  an  average  of  43  days  from  receipt  of  the 
proposal  until  the  contractor  is  authorized  to  proceed. 

This  time  is  for  reviewing  the  ECP,  planning  its  implementa¬ 
tion,  and  modifying  the  ECP  if  necessary. 

For  the  supplemental  agreement,  the  average  time 
required  to  "turn  on"  the  contractor  from  the  time  the  pro¬ 
gram  office  received  the  ECP  was  186  days.  The  primary  dif¬ 
ference  in  time  is  attributed  to  the  time  the  contractor 
spends  preparing  the  fully  priced  proposal  and  the  time  that 
both  the  government  and  contractor  spend  negotiating  an 
equitable  adjustment.  This  study  concludes  that  the  change 
order  is  the  more  expedient  of  the  two  alternatives.  These 
findings  are  consistent  with  those  reported  in  the  litera¬ 
ture  [4:110] . 

The  second  objective  of  this  study  was  to  test  a 
model  to  assist  program  managers  in  determining  whether  an 
ECP  should  be  incorporated  by  change  order  or  by  supplemental 
agreement.  Three  methods  were  used  to  test  the  validity  of 
the  model:  case  analysis,  correlation  analysis,  and  algebraic 
analysis.  The  results  of  all  three  methods  were  inconclu¬ 
sive.  First,  the  conclusions  generated  by  case  analysis  are 
based  on  too  small  a  sample.  Second,  while  the  correlation 
analysis  indicated  a  very  strong  correlation  between  actual 
and  predicted  results,  there  was  no  way  to  distinguish  be¬ 
tween  how  much  of  the  correlation  was  due  to  the  validity 
»f  the  model,  and  how  much  was  due  to  the  fact  that  both  the 
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actual  and  predicted  results  are  directly  proportional  to 
the  not-to-exceed  price.  Finally,  algebraic  analysis  was 
inconclusive  because  there  was  no  way  to  determine  if  the 
model's  preference  for  change  orders  was  invalid,  or  if  the 
assumed  loss  of  cost  control  associated  with  change  orders 
was  overstated. 

The  fact  that  the  tests  conducted  in  Phase  II  of 
this  study  were  inconclusive  does  not  rule  out  the  use  of 
the  model  by  other  programs.  Ultimately,  the  validity  of 
the  model  is  dependent  upon  the  accuracy  of  the  estimated 
price  adjustment  factors  for  change  orders  and  supplemen¬ 
tal  agreements  as  well  as  the  estimated  time  delay.  A  pro¬ 
gram  which  can  formulate  accurate  estimates  for  these  para¬ 
meters,  can  use  this  model  very  successfully.  Finally,  it 
should  be  noted  that  the  model  should  only  be  applied  in 
those  instances  where  cost  is  driving  the  decision  between 
the  two  alternatives.  When  there  are  safety  factors  in¬ 
volved  the  urgency  of  the  problem  may  require  the  use  of  a 
change  order  regardless  of  the  cost  impact. 

The  major  limitation  of  this  study  stems  from  the 
framework  employed  to  compare  the  change  order  to  the  supple¬ 
mental  agreement.  Change  orders  result  from  ECPs  submitted 
with  only  an  NTE  price.  Supplemental  agreements  usually  in¬ 
corporate  ECPs  submitted  with  a  firm  quote.  The  only  way 
to  insure  a  valid  comparison  was  to  limit  the  study  to  sup¬ 
plemental  agreements  which  incorporated  ECPs  originally 
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submitted  with  a  NTE  price.  Supplemental  agreements  which 
incorporated  ECPs  which  were  originally  submitted  with  a 
firm  price  were  excluded  from  this  study.  It  is  probable 
that  over  the  life  of  the  system,  the  use  of  the  supplemen¬ 
tal  agreement  is  less  expensive.  The  recommendations  for 
further  study  will  include  a  recommendation  to  test  this 
hypothesis . 

Despite  this  limitation,  it  is  clear  that  at  least 
in  the  case  of  the  A-10A  program,  when  a  contractor  sub¬ 
mitted  an  urgent  ECP  with  a  not-to-exceed  price,  the  change 
order  proved  to  be  the  least  costly  instrument  to  incorpor¬ 
ate  the  ECP  into  the  existing  contracts  (a  =  .25).  Efforts 
which  were  delayed  to  allow  for  incorporation  by  supplemen¬ 
tal  agreement  in  accordance  with  current  procurement  policy 
ultimately  resulted  in  a  greater  expense  to  the  government. 
Furthermore,  the  change  order  was  considerably  more  expedi¬ 
ent  than  the  supplemental  agreement.  Since  other  programs 
may  face  similar  acquisition  environments,  this  study  sug¬ 
gests  that  change  order  policy  be  administered  on  a  system- 
by-system  basis.  This  study  can  draw  no  conclusions  con¬ 
cerning  efforts  which  were  slated  for  incorporation  by  sup¬ 
plemental  agreement  from  their  inception. 

Recommended  Further  Study 

1.  This  study  did  not  assess  the  cost  impact  of 
supplemental  agreements  which  were  originally  submitted  as 


55 


a  fully  priced  proposal.  A  study  should  be  performed  which 
would  investigate  contracts  which  have  already  been  termi¬ 
nated  and  determine  what  percentage  of  the  contract  modifi¬ 
cations  were  change  orders  and  what  percentage  of  the  con¬ 
tract  modif ications  were  supplemental  agreements.  The  study 
would  then  determine  the  cost  growth  which  was  experienced 
over  the  life  of  the  contract.  If  this  information  was 
taken  for  a  number  of  contracts  the  study  could  determine 
if  the  cost  growth  per  contract  was  a  function  of  the  type 
of  contract  modifications  which  were  issued.  The  study 
should  hypothesize  that  the  contracts  with  the  highest  per¬ 
centage  of  supplemental  agreements  would  evidence  the  least 
cost  growth. 

2.  This  study  suggested  that  the  change  order 
could  be  a  very  useful  tool  even  in  an  ordinary  situation. 
One  of  the  factors  affecting  the  use  of  the  change  order  is 
the  "timeline"  management  system  which  is  imposed  when  an 
effort  is  incorporated  by  change  order.  Essentially,  this 
system  requires  that  a  change  order  be  definitized  within 
180  days  from  the  date  the  modification  is  approved  and 
issued.  The  timely  def initization  of  cnange  orders  is  scru¬ 
tinized  at  all  levels  within  the  procurement  system  and  in 
fact  the  pressure  to  definitize  the  change  order  quickly  can 
adversely  impact  a  negotiator's  leverage.  A  study  should  be 
performed  which  would  analyze  "time-line"  management  proce¬ 
dures  currently  in  effect  with  a  special  emphasis  on  incor- 
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porating  procedures  which  would  allow  the  negotiator  to  ex¬ 
ceed  the  180  day  limit  in  cases  where  the  negotiator  feels 
the  time  limit  is  adversely  impacting  his  leverage.  The 
waivers  could  be  controlled  to  prevent  the  level  of  unde- 
finiticed  work  from  getting  out  of  control.  Additionally, 
the  very  existence  of  these  "waiver"  procedures  would  pre¬ 
vent  the  contractor  from  ever  assuming  the  negotiator  is 
running  out  of  time,  and  consequently,  the  procedures  would 
help  even  in  cases  where  the  waivers  were  not  granted. 

3.  This  study  was  performed  on  the  A-10A  weapon 
system.  This  system  is  an  aircraft  system  which  consists 
of  both  production  and  retrofit  units.  The  system  is  pro¬ 
duced  by  the  Fairchild  Republic  Company.  The  data  used  for 
this  analysis  was  taken  from  the  very  late  stages  of  produc¬ 
tion.  This  study  should  be  repeated  for  programs  which 
alter  any  of  the  variables  described  above.  The  study 
should  repeat  both  Phase  I  and  Phase  II  for  these  additional 
systems.  The  results  of  the  additional  studies  should  then 
be  compared  and  integrated  with  the  results  of  this  study. 

If  the  studies  conducted  on  additional  systems  generate  d 
similar  results,  there  would  be  a  strong  argument  to  de- 
emphasize  the  pressure  exerted  against  the  use  of  the  change 
order . 
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ECP 

CONTRACT 

MOD  # 

MOD  # 

F0270-5  Rl 

F33657-73-C-0135 

P00232 

(C.O.) 

P00246 

(S/A) 

F0465  &  -1 

F33657-79-C-0502 

P00035 

(S/A) 

F0477R1  &  - 

1R1  -0135 

P00209 

(S/A) 

F0477R1 

-0502 

P00058 

(S/A) 

F0524  Rl 

-0135 

P00163 

(C.O. ) 

P00207 

(S/A) 

F0548P 

-0502 

P00060 

(S/A) 

F0550 ,  R2 

-0135 

P00166 

(Long 

Lead) 

P00199 

(C.O. ) 

P00228 

(S/A) 

F0570  &  -1 

-0502 

P00052 

(S/A) 

F0604 

-0135 

P00181 

(C.O.) 

P00221 

(S/A) 

F0604 

-0502 

P00033 

(C.O. ) 

P00057 

(S/A) 

F0610P 

-0135 

P00217 

(C.O.) 

P000268 

(S/A) 

F0610P 

-0502 

P00065 

(C.O.) 

P00110 

(S/A) 

F0621 

-0502 

P00074 

(C.O. ) 

P00114 

(S/A) 

F0905 

-0502 

P00048 

(C.O.) 

P00085 

(S/A) 

F0908P 

-0135 

P00184 

(C.O.) 

P00216 

(S/A) 

F0908 

-0502 

P00053 

(S/A) 

F0910P 

-0135 

P00213 

(C.O.) 

P00268 

(S/A) 

F0910P 

-0502 

P00062 

(C.O.) 

P00110 

(S/A) 

F2206R3-1R3 

-0135 

P00209 

(S/A) 

F2206R3 

-0502 

P00058 

(S/A) 

F2768R1 

-0135 

P00222 

(S/A) 

F2768R1 

-0502 

P00069 

(S/A) 

F2780  &  Rl 

-0135 

P00194 

(S/A) 

F2780  &  Rl 

-0502 

P00042 

(S/A) 

F2798 

-0135 

P00233 

(C.O.) 

P00265 

(S/A) 

F2798 

-0502 

P  0007  5 

(C.O.) 

P00109 

(S/A) 

F3055R4 

-0135 

P00234 

(S/A) 

F3151R1 

-0135 

P00176 

(C.O.) 

P00209 

(S/A) 

F3151R1 

-0502 

P00026 

(C.O.) 

P00058 

(S/A) 

F3156R1 

-0135 

P00194 

(S/A) 

F3156R1 

-0502 

P00042 

(S/A) 

F3159  R2 

-0502 

P00027 

(C.O. ) 

P00071 

(S/A) 

F3194 

-0135 

P00256 

(S/A) 

F3195 

-0135 

P00163 

(C.O.) 

P00207 

(S/A) 

F3195 

-0502 

P00015 

(C.O.  ) 

P00059 

(S/A) 

F3235 

-0135 

P00139 

(C.O. ) 

P00192 

(S/A) 

F3235R1P 

-0135 

P00217 

(C.O. ) 

P00268 

(S/A) 

F3235R1 

-0502 

P00165 

(C.O. ) 

P00110 

(S/A) 

F3246  Rl  & 

1R1  -0502 

P00110 

(S/A) 

F3251 

-0502 

P00063 

(S/A) 

F3253 

-0135 

P00203 

(S/A) 

F3255 

-0135 

P00148 

(C.O. ) 

P00192 

(S/A) 
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ECP 

CONTRACT 

MOD  # 

MOD 

# 

F3272 

-0135 

P00169 

(Long  Lead) 

P00196 

(C.O.) 

P00236 

(S/A) 

F3272 

-0502 

P00044 

(C.O. ) 

P00085 

(S/A) 

F3280 

-0502 

P00085 

(S/A) 

F3324R2P 

-0135 

P00248 

(C.O. ) 

P00290 

(S/A) 

F3324R2 

-0502 

P00091 

(C.O. ) 

L00126 

(S/A) 

F3325R2P 

-0135 

P00253 

(C.O. ) 

P00271 

(S/A) 

F3325P 

-0502 

P00123 

(C.O. ) 

F3336P 

-0135 

P00273 

(C.O.) 

P00305 

(S/A) 

F3336P 

-0502 

P00118 

(C.O.) 

P00170 

(S/A) 
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CHANGE  ORDERS 
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INDEX 


ECP 


CONTRACT 


01 

F0270-5R1 

-0135 

02 

F0524  R1 

-0135 

03 

F0550,  R2 

-0135 

04 

F0604 

-0135 

05 

F0604 

-0502 

06 

F9610P 

-0135 

07 

F0610P 

-0502 

08 

F0621 

-0502 

09 

F0905 

-0502 

10 

F0908P 

-0135 

11 

F0910P 

-0135 

12 

F0910P 

-0502 

13 

F2798 

-0135 

14 

F2798 

-0502 

15 

F3151R1 

-0135 

16 

F3151R1 

-0502 

17 

F3159R2 

-0502 

18 

F3195 

-0135 

19 

F3195 

-0502 

20 

F3235 

-0135 

21 

F3235R1P 

-0135 

22 

F3235R1 

-0502 

23 

F3272 

-0135 

24 

F3272 

-0502 

25 

F3324R2P 

-0135 

26 

F3324R2 

-0502 

27 

F3336P 

-0135 

28 

F3336P 

-0502 

*Four  change  orders  were  deleted  from  the  original 

list  of  51  for  the  following  reasons: 

1.  F0548  -  the  cost  information  for  the  initial 
change  order  was  unavailable. 

2.  F3255  -  the  recommended  fix  was  redefined  after 
the  issuance  of  the  initial  change  order,  con¬ 
sequently  the  definitized  price  does  not  address 
the  same  level  of  effort  as  the  initial  not-to- 
exceed  price. 

3.  F3325R2P  and  F3325P  -  these  efforts  were  termi¬ 
nated  prior  to  completion. 
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SUPPLEMENTAL  AGREEMENTS 
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INDEX 


ECP 


CONTRACT  # 


29 

F0477R1-1R1 

F0477R1 

-0135 

-0502 

30 

F0908 

-0502 

31 

F2206R3  &  1R3 
F2206  R3 

-0135 

-0502 

32 

F2768R1 

F2768R1 

-0135 

-0502 

33 

F2780  &  Rl 

F2780  &  Rl 

-0135 

-0502 

34 

F3055R4 

-0135 

35 

F3156R1 

F3156R1 

-0135 

-0502 

36 

F3251 

-0502 

37 

F3253 

-0135 
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tHa>  CRITICAL 

(H  )  ALTERNATE  VALUE  TEST 

TEST  #  HYPOTHESIS  HYPOTHESIS  (a  =  .05)  STATISTIC  CONCLUSION 


Note : 


APPENDIX  E 


PRICE  ADJUSTMENT  -  CHANGE  ORDERS 


INDEX 


*  1 


(NOT-TO-EXCEED 

PRICE) 


Pf 

(DEFINITIZED 

PRICE) 


PAF 

c 


01 

-.,  256,065 

3,325,000 

21.0 

02 

560,000 

354,000 

36.7 

03 

11,794,900 

8,847,943 

25.0 

04 

793,000 

780,305 

1.6 

05 

70,000 

44,928 

35.8 

06 

593,300 

371,627 

37 . 4 

07 

426,800 

33,824 

21.8 

08 

164 ,400 

106,117 

35.5 

09 

246,900 

164,790 

33.3 

10 

126,800 

120,384 

5.1 

11 

103,600 

60,341 

41.8 

12 

213,100 

151,577 

28 . 9 

13 

26,700 

26,700 

0.0 

14 

21,300 

16,559 

22.3 

15 

170,600 

64,817 

62.0 

16 

195,500 

185,452 

5.1 

17 

1,991,500 

1,201,700 

39.7 

18 

832,900 

658,500 

20.9 

19 

458,000 

436,500 

4 . 7 

20 

1,543,100 

1,069,014 

30.7 

21 

585,200 

389,923 

33.4 

22 

593,600 

440,804 

25.7 

23 

606,700 

395,759 

34.8 

24 

327,800 

103,597 

68.4 

25 

253,000 

164,713 

34.9 

26 

185,900 

114,786 

38.2 

27 

132,240 

97,385 

26.4 

28 

134,900 

64,627 

52.1 

PAFC  =  29.432% 


95%  C.I.  =  23.01  <  m  <  35.85 
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APPENDIX  F 

PRICE  ADJUSTMENT  -  SUPPLEMENTAL  AGREEMENTS 
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INDEX 


[DEX 

ps 

NOT-TO-EXCEED 

PRICE 

Pf 

(DEFINITIZED 

PRICE) 

PAFS 

29 

138,300 

93,714 

32.3 

30 

39,000 

34,656 

11.1 

32 

666,833 

476,492 

28.5 

33 

268,400 

173,334 

35.4 

34 

21,600 

18,641 

13.7 

35 

168,700 

135,574 

19.6 

36 

32,817 

24,951 

24 . 0 

37 

38,200 

29,571 

22.6 

PAFg  =  23.287% 


95%  C.I.  =  16.34  <  *i  <  30.23 
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APPENDIX  G 

PROCESSING  TIME  -  CHANGE  ORDERS 
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INDEX  # 

Tb 

DATE 

ECP  REC’D 

SEQ 

VALUE 

Tf 

EFFECTIVE 

DATE 

OF  MOD 

SEQ 

VALUE 

Tc 

PROCESS 

TIME 

01 

2 

Apr 

81 

489 

19 

Jun 

81 

567 

78 

02 

22 

Jul 

80 

235 

15 

Oct 

80 

320 

85 

03 

12 

Nov 

80 

348 

20 

Feb 

81 

448 

100 

04 

17 

Nov 

80 

353 

3 

Dec 

80 

369 

16 

05 

17 

Nov 

80 

353 

3 

Dec 

80 

369 

16* 

06 

18 

Mar 

81 

473 

11 

May 

81 

528 

55 

07 

18 

Mar 

81 

473 

11 

May 

81 

528 

55* 

08 

21 

May 

81 

538 

12 

Jun 

81 

560 

22 

09 

29 

Dec 

80 

395 

5 

Mar 

81 

461 

66 

10 

1 

Dec 

80 

367 

23 

Dec 

80 

389 

22 

11 

14 

Apr 

81 

501 

27 

Apr 

81 

514 

13 

12 

14 

Apr 

81 

501 

27 

Apr 

81 

514 

13* 

13 

17 

Jun 

81 

565 

19 

Jun 

81 

567 

02 

14 

17 

Jun 

81 

565 

19 

Jun 

81 

567 

02* 

15 

15 

Oct 

80 

320 

17 

Nov 

80 

353 

33 

16 

15 

Oct 

80 

320 

18 

Nov 

80 

354 

34* 

17 

1 

Oct 

80 

306 

17 

Nov 

80 

353 

47 

18 

18 

Aug 

80 

262 

15 

Oct 

80 

320 

58 

19 

18 

Aug 

80 

262 

15 

Oct 

80 

320 

58* 

20 

27 

Jun 

80 

210 

3 

Jul 

80 

216 

06 

21 

7 

May 

81 

524 

11 

May 

81 

528 

04 

22 

12 

Aug 

81 

621 

24 

Nov 

81 

725 

104 

23 

17 

Dec 

80 

383 

6 

Feb 

81 

434 

51 

24 

17 

Dec 

80 

383 

6 

Feb 

81 

434 

51* 

25 

20 

Aug 

81 

629 

21 

Aug 

81 

630 

01 

26 

9 

Dec 

81 

740 

4 

Feb 

82 

797 

57 

27 

3 

Nov 

81 

704 

14 

Dec 

81 

745 

41 

28 

3 

Nov 

81 

704 

14 

Dec 

81 

745 

41* 

♦Indicates  points  which  were  not  included  in  the  calcula¬ 
tion  of  mean  processing  time.  These  sample  points 
result  from  the  same  ECPs  as  the  points  immediately 
above  them  in  the  table. 


T  =  43.05  days 

c 


95%  C.I. 


27.88  <  u  <  58.21 
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PROCESSING  TIME  -  SUPPLEMENTAL  AGREEMENTS 
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INDEX  # 

Tb 

DATE 

ECP  REC'D 

SEQ 

VALUE 

Tf 

EFFECTIVE 

DATE 

OF  MOD 

SEQ 

VALUE 

Tc 

PROCESS 

TIME 

29 

7 

Dec 

79 

07 

23 

Apr 

80 

145 

138 

30 

1 

Dec 

80 

367 

6 

Apr 

81 

493 

126 

31 

10 

Jun 

80 

193 

23 

Apr 

81 

510 

317 

32 

20 

Aug 

80 

264 

22 

Jun 

81 

570 

306 

33 

25 

Jun 

80 

208 

17 

Feb 

81 

445 

237 

34 

22 

Jun 

81 

570 

30 

Jun 

81 

578 

08 

35 

4 

Sep 

80 

279 

17 

Feb 

81 

445 

166 

36 

31 

Oct 

80 

336 

8 

May 

81 

525 

189 

37 

17 

Sep 

80 

292 

24 

Mar 

81 

480 

188 

Ts  =  186.11 

95%  C.I.  =  114.50  <  u  <  257.72 
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APPENDIX  I 
CASE  ANALYSIS  DATA 
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ESTIMATED 


78 


digit  numbers  in  parenthesis  identify  contracts.  Some  of  th 
listed  in  the  chart  are  different  for  each  contract  affected 
s  ECP .  In  these  cases,  both  values  used  have  been  presented 


0) 

* 

X 

•O 

■p 

a) 

■p 

c 

(1)  T3  *p 

a)  -h 

tn  a)  io 

x.  £ 

a)  tn  p 

Eh  p 

X  tn  o 

as 

Eh  <V  P 

■u 

P  -H 

•  as 

TJ  Its 

in  T3 

•  T3 

n 

in  as  as 

P  OS 

as  X 

o  ,Q 

>  x  x> 

l 

•P  o 

O 

-p  -P  <P 

« -u 

ns  X,  0 

• 

c  3 

as  -u 

p  -p 

•  4) 

as  tn  p 

•P  >, 

P  c  ns 

ip  O  Oi 

to 

ItS  *P 

4->  -P 

-p  4: 

O 

O  ftO 

itJ  Pi 

3  O  -p 

Pi  q) 

-p  £ 

•P 

as  3 

c  e 

0  -P 

O  3 

-p  .a  c 

O  G 

nS  *P 

tn  tn 

OS  -P 

C-P  tn 

-P  O 

■p  0  as 

ttS  nS 

US  P  -P 

-P  P 

POP 

P  -P 

P  X 

cu  c 
o  o 
p  o 


rr  m 

E 

<M  OS 
XI  -U 
Eh  -H 


X 
•H 
OS  OS  IP 
O 

>1  c 


a 

x  p  as 

-P 

a  as 

0 

as  p 

ns  X 

*p 

+j  as 

tn 

-p 

JG 

ns  oh 

as 

as 

3 

p  ip 

tn 

X  tn 

ns  -p 

0 

•p  as 

nS 

CU-C 

a 

-p 

■P 

as 

0 

X  ns 

ns 

as  as 

p 

-p  0 

•H  *H 

X 

as  Eh 

a 

3  x> 

in 

p 

as 

c 

■P 

as 

> 

-U  -P 

C 

3  • 

•H 

as 

•H  « 
>P  Q 
-P  CQ 
-P  Eh 
C  — 
<D 

m'O  h 
£'H  O 

•u  xt 
n  e 
>i  c  >i 
xx  ns  m 


M  4J 

£  ns 
C 
p 

a) 

o  -P 
P  P 

a  ns 


as 

us  tn 
as  as  as 
P  > 

Dii-H  XX 

-P  ‘ 

M  (0 

M  C 
P 

OS  OS  OS  JS 
tn 
(0 


E  o 

nS  (0 


u  nj  33  0) 
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♦SAMPLE  Xi  Yi  REVISED 

INDEX  NUMBER  ACTUAL  PRICE  ESTIMATED  PRICE  ESTIMATE 


01 

3,325,000 

3,003,420 

02 

354,000 

395,180 

03 

8,947,943 

8,323,425 

04 

780,305 

559,604 

05 

44,928 

49,398 

06 

371,627 

418,680 

07 

333,824 

310,390 

08 

106,117 

116,104 

09 

164,790 

174,232 

10 

120,384 

92,215 

w  11 

60,341 

75,343 

£  12 

151,577 

150,380 

9  13 

26,700 

19,417 

O  14 

16,559 

15,490 

M  15 

64,817 

120,389 

u  16 

185,452 

142,177 

<  17 

1,201,700 

1,448,318 

g  18 

658,500 

605,727 

19 

436,500 

323,201 

20 

1,069,014 

1,122,219 

21 

389,923 

425,587 

22 

440,804 

431,696 

23 

395,759 

441,223 

24 

103,597 

231,322 

25 

164,713 

183,994 

26 

114,786 

131,186 

27 

97,385 

93,319 

28 

64,627 

98,106 

►4 

<  cn 

29 

30 

93, 714 

34 , o56 

106,090 

29,879 

106,105 

29,879 

E-  E-1 
2  2 

31 

- 

- 

- 

w  eg 

32 

351,174 

382,232 

387,138 

w  u 

33 

173,334 

210,028 

211,760 

n  w 

(X  K 

34 

18,641 

16,548 

16,548 

(X  U 
D  < 
W 

35 

135,574 

116,216 

113,822 

36 

24,951 

24,709 

24,753 

37 

29,571 

29,266 

29,266 

♦These  index  numbers  correspond  to  the  index  numbers  in 
Appendix  B  and  Appendix  C 
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